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1 Introduction 

The isomorphism problem for finitely generated Coxeter groups is the prob- 
lem of deciding if two finite Coxeter matrices define isomorphic Coxeter 
groups. Coxeter [3] solved this problem for finite irreducible Coxeter groups. 
Recently there has been considerable interest and activity on the isomor- 
phism problem for arbitrary finitely generated Coxeter groups. 

In this paper we describe a family of isomorphism invariants of a finitely 
generated Coxeter group W. Each of these invariants is the isomorphism 
type of a quotient group W/N of W by a characteristic subgroup N. The 
virtue of these invariants is that W/N is also a Coxeter group. For some 
of these invariants, the isomorphism problem of W/N is solved and so we 
obtain isomorphism invariants that can be effectively used to distinguish 
isomorphism types of finitely generated Coxeter groups. 

We emphasize that even if the isomorphism problem for finitely generated 
Coxeter groups is eventually solved, several of the algorithms described in our 
paper will still be useful because they are computational fast and would most 
likely be incorporated into an efficient computer program that determines if 
two finite rank Coxeter systems have isomorphic groups. 

In §2, we establish notation. In §3, we describe two elementary quoti- 
enting operations on a Coxeter system that yields another Coxeter system. 
In §4, we describe the binary isomorphism invariant of a finitely generated 
Coxeter group. In §5, we review some matching theorems. In §6, we describe 
the even isomorphism invariant of a finitely generated Coxeter group. In 
§7, we define basic characteristic subgroups of a finitely generated Coxeter 
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group. In §8, we describe the spherical rank two isomorphism invariant of a 
finitely generated Coxeter group. In §9, we make some concluding remarks. 

2 Preliminaries 

A Coxeter matrix is a symmetric matrix M = (m(s,t)) Stte s with m(s,t) 
either a positive integer or infinity and m(s,t) = 1 if and only if s — t. 
A Coxeter system with Coxeter matrix M = (m(s,t)) s> tes is a pair (W, S) 
consisting of a group IV and a set of generators S for IV such that IV has 
the presentation 

W = (S | (st) m(s ' 4) : s,t e S and m(s,t) < oo). 

We call the above presentation of W, the Coxeter presentation of (W, S). If 
(W, S) is a Coxeter system with Coxeter matrix M = (m(s, t)) s ^ s , then the 
order of st is m(s,t) for each s, t in S by Prop. 4, p. 92 of Bourbaki [3], 
and so a Coxeter system (W, S) determines its Coxeter matrix; moreover, 
any Coxeter matrix M = (m(s,t)) Stt ^s determines a Coxeter system (IV, S) 
where W is defined by the corresponding Coxeter presentation. If (W, S) is 
a Coxeter system, then W is called a Coxeter group and S is called a set 
of Coxeter generators for IV, and the cardinality of S is called the rank of 
(W, 5). A Coxeter system (W, S) has finite rank if and only if W is finitely 
generated by Theorem 2 (iii), p. 20 of Bourbaki 

Let (IV, S) be a Coxeter system. A visual subgroup of (IV, S) is a subgroup 
of IV of the form (A) for some A C S. If (A) is a visual subgroup of (IV, S), 
then ((A), A) is also a Coxeter system by Theorem 2 (i), p. 20 of Bourbaki 

m 

When studying a Coxeter system (IV, S) with Coxeter matrix M it is 
helpful to have a visual representation of (W,S). There are two graphical 
ways of representing (IV, S) and we shall use both depending on our needs. 

The Coxeter diagram (C-diagram) of (W, S) is the labeled undirected 
graph A = A (IV, S) with vertices S and edges 

{(s, t) : s,t e S and m(s,t) > 2} 

such that an edge (s, t) is labeled by m(s, t). Coxeter diagrams are useful for 
visually representing finite Coxeter groups. If A C S, then A((A),A) is the 
sub diagram of A (IV, S) induced by A. 
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A Coxeter system (W, S) is said to be irreducible if its C-diagram A is 
connected. A visual subgroup (A) of (W, S) is said to be irreducible if ((^4), ^4) 
is irreducible. A subset A of S is said to be irreducible if (A) is irreducible. 

A subset A of S is said to be a component of S if A is a maximal irreducible 
subset of S or equivalently if A((A), A) is a connected component of A(W, S). 
The connected components of the A(W, S) represent the factors of a direct 
product decomposition of W. 

The presentation diagram (P-diagram) of (W, S) is the labeled undirected 
graph T = T(W, S) with vertices S and edges 

{(s, t) : s,t E S and m(s,t) < 00} 

such that an edge (s, t) is labeled by m(s, t). Presentation diagrams are useful 
for visually representing infinite Coxeter groups. If A C S, then r((A),.A) 
is the subdiagram of T(W, S) induced by A. The connected components of 
T(W, S) represent the factors of a free product decomposition of W. 

For example, consider the Coxeter group W generated by the four reflec- 
tions in the sides of a rectangle in E 2 . The C-diagram of (W, S) is the disjoint 
union of two edges labeled by 00 while the P-diagram of W is a square with 
edge labels 2. 

Let (W, S) and (W, S') be Coxeter systems with P-diagrams F and T', 
respectively. An isomorphism <ft : (W, S) — > (W, S') of Coxeter systems is an 
isomorphism : W — > W such that <fi(S) = S'. An isomorphism ip : T — ► T' 
of P-diagrams is a bijection from S to S" that preserves edges and their labels. 
Note that (W, 5) £ (W, S") if and only if T = T'. 

We shall use Coxeter's notation on p. 297 of [H] for the irreducible spher- 
ical Coxeter simplex reflection groups except that we denote the dihedral 
group by D 2 (&). Subscripts denote the rank of a Coxeter system in 
Coxeter's notation. Coxeter's notation partly agrees with but differs from 
Bourbaki's notation on p. 193 of 0. 

Coxeter j3] proved that every finite irreducible Coxeter system is isomor- 
phic to exactly one of the Coxeter systems A n , n > 1, B n , n > 4, C n , n > 2, 
D 2 (/c), k > 5, E 6 , E 7 , E 8 , F 4 , G3, G4. For notational convenience, we define 
B 3 = A 3 , D 2 (3) = A 2 , and D 2 (4) = C 2 

The type of a finite irreducible Coxeter system (W, S) is the isomorphism 
type of (W, S) represented by one of the systems A n , B n , C n , D 2 (&), E 6 , 
E 7 , E 8 , F 4 , G 3 , G 4 . The type of an irreducible subset A of S is the type of 
((A), A). 
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The C-diagram of A n is a linear diagram with n vertices and all edge 
labels 3. The C-diagram of B ra is a Y-shaped diagram with n vertices and 
all edge labels 3 and two short arms of consisting of single edges. The C- 
diagram of C n is a linear diagram with n vertices and all edge labels 3 except 
for the last edge labelled 4. The C-diagram of D 2 (fc) is a single edge with 
label k. The C-diagrams of E 6 , E 7 , E 8 are star shaped with three arms and 
all edge labels 3. One arm has length one and another has length two. The 
C-diagram of F4 is a linear diagram with edge labels 3, 4, 3 in that order. The 
C-diagram of G3 is a linear diagram with edge labels 3, 5. The C-diagram of 
G4 is a linear diagram with edge labels 3, 3, 5 in that order. 

3 Elementary Quotient Operations 

In this section we describe two types of elementary edge quotient opera- 
tions on a Coxeter system (W, S) of finite rank. The first we call edge label 
reduction and the second we call edge elimination. 

Suppose s and t are distinct elements of S with m(s,t) < 00. Let d be a 
positive divisor of m — m(s, t), with d < m, and let N be the normal closure 
of the element (st) d of W. Then a presentation for W/N is obtained from the 
Coxeter presentation for (W, S) by adding the relator (st) d . Asm = (m/d)d, 
the relator (st) m is derivable from the relator (st) d and so the relator (st) m 
can be removed from the presentation for W/N. 

Assume d > 1. Then the presentation for W/N is a Coxeter presentation 
whose P-diagram is obtained from the P-diagram for (W, S) by replacing the 
label m on the edge (s, t) with the label d. We call the operation of passing 
from the Coxeter system (W, S) to the quotient Coxeter system (W/N, {sN : 
s G S}) the (s, t) edge label reduction from m to d. For example, if we reduce 
the 4 edge of F 4 to 2, we obtain the Coxeter system A 2 x A 2 . 

Now assume d — 1. We delete from the presentation for W/N the gen- 
erator t and the relator st and replace all occurrences of t in the remaining 
relators by s. Suppose r is in S and k = m(r, s) < 00 and £ = m(r, t) < 00. 
Then we have the relators (rs) k and (rs) e in the presentation for W/N. Let 
d be the greatest common divisor of k and £. Then there are integers a and 
b such that d = ak + hi. This implies that (rs) d is derivable from (rs) k and 
(rs) e and so we may add the relator (rs) d to the presentation for W/N. Then 
(rs) k and (rs) e are derivable from (rs) d and so we can eliminate the relators 
(rs) k and (rs) 1 from the presentation for W/N. We do this for each r in S 
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such that m(r,s) < oo and m(r,t) < oo. On the P-diagram level, we have 
eliminated the edge (s, t) and identified the vertices s and t and we have 
coalesced each edge (r, s) with label k < oo with the edge (r, t) with label 
£ < oo to form an edge with label d the greatest common divisor of k and £. 
If each common divisor d is greater than one, we obtain a Coxeter presenta- 
tion for W/N. If some common divisor d is one, we delete the corresponding 
generator r and repeating the above reduction procedure on the presentation 
of W/N. As the set S of generators is finite, we will eventually stop deleting 
generators and obtain a Coxeter presentation for W/N with generators the 
subset S' of {sN : s E S} corresponding to the undeleted elements of S. We 
call the operation of passing from the Coxeter system (W, S) to the quotient 
Coxeter system (W/N,S') the (s,t) edge elimination. For example, if we 
eliminate the 3 edge from C3, we obtain the Coxeter system Ai x Ai. 

4 The Binary Isomorphism Invariant 

Let (W : S) be a Coxeter system of finite rank. For each pair of elements s, t 
of S with m(s, t) < 00, let b(s, t) be the 2-part of m{s, t), that is, b(s, t) is the 
largest power of 2 that divides m(s, t). If m(s, t) = 00, we define b(s, t) = 00. 
Let Nb be the normal closure in W of all the elements of the form (st) b ^ s '^ 
with b(s,t) < 00, and let W b = W/N b . Let 77 : W — > W b be the quotient 
homomorphism, and let S b = 7)(S). 

Proposition 4.1 The pair (W b , S b ) is a Coxeter system such that if s and t 
are in S , then r)(s) = i](t) if and only if s and t are conjugate in W . If s and 
t are nonconjugate elements of S, then the order of r)(s)r)(t) is the minimum 
of the set of all b(u, v) such that u and v are in S and u is conjugate to s 
and v is conjugate to t. In particular, the order of r)(s)r)(t) is a power of 2 
or 00. 

Proof: The system (W b , S b ) can be obtained from (W, S) by a sequence of 
elementary quotient operations. First we can do a series of edge label reduc- 
tions of all the even labelled edges of the P-diagram of (W, S) to their 2-parts. 
Then we do a series of edge eliminations of all the odd labelled edges. Each 
element of the form (st) b( - s '^ with b(s,t) < 00 is in the commutator subgroup 
of W. Therefore abelianizing W factors through the quotient W/N b , and 
so r](s) = r)(t) if and only if s and t are the same odd component of the 
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P-diagram of (W, S). Hence r](s) = rj(t) if and only if s and t are conjugate 
in W by Prop. 3, p. 12 of Bourbaki 

Suppose s and t are nonconjugate elements of S and u and v are in S, 
with m(u,v) < oo, and u is conjugate to s and v is conjugate to t. Then 
u and f are not conjugate, and so m(u, v) is even, and therefore b(u, v) is a 
power of 2 greater than 1. In the coalescence of two such edges, the greatest 
common divisor is the minimum of the two edge labels. Therefore the order 
of r)(s)rj(t) is the minimum of the set of all b(u,v) such that u and v are in 
S and u is conjugate to s and v is conjugate to t. □ 



Theorem 4.2 Let (W, S) be a Coxeter system of finite rank. For each pair 
of elements s,t of S with m(s,t) < oo, let b(s,t) be the largest power of 2 
that divides m(s,t). Let Nb be the normal closure in W of all the elements 
of the form (st) b ( s ' t ' ) with m(s,t) < oo, Then iV 6 is the normal closure in W 
of the set of all elements of W of odd order. Therefore iV& is a characteristic 
subgroup ofW that does not depend on the choice of Coxeter generators S. 

Proof: Every element of the form (st) b( - s '^ with m(s, t) < oo has odd order, 
and so iVj, is contained in the normal closure of all the elements of odd order. 
Let w be an element of odd order, then r)(w) has odd order in H4 = W/N^. 
By the previous proposition, (W&, S&) is a Coxeter system with all edge labels 
a power of 2. Therefore r](w) is conjugate to an element of odd order of a 
finite visual subgroup of (W&, S&) by 0, Ch. V, §4, Ex. 2. The finite visual 
subgroups of (Wb, Sft) are direct products of groups of type Ai and C 2 , and 
so are 2-groups. Therefore Wb has no nontrivial elements of odd order. Hence 
rj(w) = 1, and so w is in JV&. Thus Nf, is the normal closure of all the elements 
of W of odd order. □ 

P. Bahls proved in his Ph.D. thesis [1] that any finitely generated Coxeter 
group has at most one P-diagram, up to isomorphism, with all edge labels 
even; see Theorem 5.2 in Bahls and Mihalik |2 . Therefore the isomorphism 
type of the P-diagram of (Wb, Sb) is an isomorphism invariant of W by The- 
orem 4.2. We call the isomorphism type of the P-diagram of (Wb, Sb) the 
binary isomorphism invariant of W. 

In Figure 1, we illustrate two P-diagrams and their binary isomorphism 
invariant P-diagrams below them. The even diagrams are not isomorphic, 
and so the top two P-diagrams represent nonisomorphic Coxeter groups. 
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Figure 1 



5 Matching Theorems 

Let (W, S) be a Coxeter system. A basic subgroup of (W, S) is a noncyclic, 
maximal, finite, irreducible, visual subgroup of (W, S). A base of (W, S) is a 
subset -B of 5* such that (B) is a basic subgroup of (W, S). The theorems in 
this section are proved in our paper [S|. 

Theorem 5.1 (Basic Matching Theorem) Let W be a finitely generated Cox- 
eter group with two sets of Coxeter generators S and S' . Let B be a base of 
(W, S). Then there is a unique irreducible subset B' of S' such that [(B), (B)} 
is conjugate to [(B'), (B')] in W . Moreover, 

1. The set B' is a base of (W, S"), 

2. If \{B)\ = \(B')\, then B and B' have the same type and there is an 
isomorphism : (B) — > (B 1 ) that restricts to conjugation on [(B), (B)] 
by an element ofW. 

3. If \(B)\ < \(B')\, then either B has type T$2q+i and B' has type C2 q +i 
for some q > 1 or B has type D 2 (2g + 1) and B' has type D 2 (4g + 2) 
for some q > 1. Moreover, there is a monomorphism <fi : (B) — > (B'} 
that restricts to conjugation on [(B), (B)} by an element ofW. 
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Let W be a finitely generated Coxeter group with two sets of Coxeter 
generators S and S'. A basic subgroup (B) of (W, S) is said to match a basic 
subgroup (B') of (W, S') if [(B), (B)] is conjugate to [(B'),(B')] in W. A 
base B of (W, 5) is said to match a base B' of (W, S") if (B) matches (B'). 

Theorem 5.2 Let (W,S) be a Coxeter system of finite rank. Let B be a 
base of (W, S) of type C 2g+ i for some q > 1, and let a, b, c be the elements 
of B such that m(a,b) = 4 and m(b,c) = 3. Then W has a set of Coxeter 
generators S' such that B matches a base B' of (W, S') of type B 2(J+1 if and 
only ifm(s, t) = 2 for all (s, t) G (S — B) x B such that m(s, a) < oo. 

Theorem 5.3 Let B be a base of (W, S) of type C 2(?+ i for some q > 1, and 
let a, b, c be the elements of B such that m(a, b) = 4 and m(b, c) = 3. Suppose 
that m(s,t) = 2 for all (s,t) G (S — B) x B such that m(s,a) < oo. Let 
d = aba, and let z be the longest element of (B) . Let S' — (S — {a}) U {d, z} 
and B' = (B — {a}) U {d}. Then S' is a set of Coxeter generators for W 
such that 

1. The set B' is a base of (W, S') of type B 2g +i that matches B, 

2. m(z,t) = 2 for all t G B' , 

3. If(s,t) G (S — B) x {d, z}, then m(s,t) < oo if and only if m(s, a) < oo, 
moreover ifm(s,t) < oo, then m(s,t) = 2. 

Theorem 5.4 Let (W, S) be a Coxeter system of finite rank, and let B = 
{a, b} be a base of (W, S) of type D 2 (4g + 2) for some q > 1. Then W has 
a set of Coxeter generators S' such that B matches a base B' of (W, S') of 
type D 2 (2g + 1) if and only if either v = a or v = b has the property that if 
s G S — B and m(s, v) < oo, then m(s, a) = m(s, b) = 2. 

Theorem 5.5 Let B = {a,b} be a base of (W,S) of type D 2 (4g + 2) for 

some q > 1. Suppose that if s G S — B and m(s,a) < oo, then m(s,a) = 
m(s,b) = 2. Let c = aba and let z be the longest element of (B) . Let 
S' — (S— {a}) U {c, z} and B' = {b, c} . Then S' is a set of Coxeter generators 
of W such that 

1. The set B' is a base of (W, S') of type D 2 (2g + 1) that matches B, 

2. m(z, b) = m(z, c) = 2, 

3. if(s, t) G (S—B) x {c, z}, then m(s, t) < oo if and only ifm(s, a) < oo ; 
moreover ifm(s,t) < oo, then m(s,t) = 2. 
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6 The Even Isomorphism Invariant 



Let (W, S) be a Coxeter system of finite rank. A pair of elements (a, b) of 
S is said to be special if (a, b) satisfy the conditions of Theorem 5.4, that 
is, if m(a, b) = 2 mod 4 and either t> = a or u = 6 has the property that if 
s G S — {a, b} and m(s, u) < oo, then m(s, a) = m(s, b) = 2. 

Let (a, b) be a pair of elements of S. If (a, 6) is special, define fn(a, b) = 2 
and if (a, 6) is nonspecial, define m(a, b) = m(a, b). 

Let N e be the normal closure in W of all the elements of the form ab 
with a and b elements of S such that m(a, b) is odd together with all the 
elements of the form (ab) 2 such that (a, b) is a special pair of elements of S. 
Let W e = W/N e , let 77 : W — > W e be the quotient homomorphism, and let 

S e = V(S). 

Proposition 6.1 The pair (W e , S e ) is a Coxeter system such that if s andt 
are in S , then i](s) = r](t) if and only if s and t are conjugate in W . If s and 
t are nonconjugate elements of S, then the order of r](s)r](t) is the greatest 
common divisor of the set of all m(u, v) such that u and v are in S and u is 
conjugate to s and v is conjugate to t. In particular, the order ofrj(s)rj{t) is 
either even or 00. 

Proof: The system (W e , S e ) can be obtained from (W, S) by a sequence 
of elementary quotient operations. First we reduce to 2 all the edge labels 
of special edges (a, b) to obtain a Coxeter system with Coxeter matrix M = 
(m(s,t)) Sj teS- Then we eliminate all the odd labelled edges. Each element 
of the form either st, with m(s,t) odd, or (st) 2 , with (s,t) special, is in the 
commutator subgroup of W. Therefore abelianizing W factors through the 
quotient W/N e , and so t](s) = r)(t) if and only if s and t are in the same odd 
component of the P-diagram of (W, S). Hence r](s) = rj(t) if and only if s 
and t are conjugate in W by Prop. 3, p. 12 of Bourbaki [3 . 

Suppose s and t are nonconjugate elements of S and u and v are in S, 
with m(u, v) < 00, and u is conjugate to s and v is conjugate to t. Then 
u and v are not conjugate, and so m(u, v) is even. In the coalescence of 
two such edges, the greatest common divisor is even. Therefore the order of 
rj(s)r](t) is the greatest common divisor of the set of all fn(u, v) such that u 
and v are in S and u is conjugate to s and v is conjugate to t. □ 
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Let (W, S) be a Coxeter system of finite rank. A base B of (W, S) is said 
to be of odd type if there are elements a and b in B, with m(a, b) odd. A base 
B of (W, S) is said to be special if B = {a, b} with (a, 6) special. 

Theorem 6.2 Let (W, S) be a Coxeter system of finite rank. Then N e is a 
characteristic subgroup of W that does not depend on the choice of Coxeter 
generators S . 

Proof: Observe that N e is the normal closure in W of the commutator 
subgroups of all the basic subgroups (B) of (W, S) such that the base B is 
either of odd type or special. Let S' be another set of Coxeter generators 
of W. By the Basic Matching Theorem and Theorem 5.4, the group N e is 
also the normal closure in W of the commutator subgroups of all the basic 
subgroups (B') of (W, S') such that the base B' is either of odd type or 
special. Therefore N e is the normal closure in W of all the elements of the 
form s't' with s' and t' in 5" and m(s', t') odd together with all the elements 
of the form (s't') 2 with (s',f) a special pair of elements of S'. Thus N e is a 
characteristic subgroup of W that does not depend on the choice of Coxeter 
generators S. □ 

Let (W, S) be a Coxeter system of finite rank. P. Bahls proved in his 
Ph.D. thesis pQ that any finitely generated Coxeter group has at most one 
P-diagram, up to isomorphism, with all edge labels even. Therefore the 
isomorphism type of the P-diagram of (W e , S e ) is an isomorphism invariant 
of W by Theorem 6.2. We call the isomorphism type of the P-diagram 
of {We-, S e ) the even isomorphism invariant of W . For example, the even 
isomorphism invariant of the system D2(6) is the isomorphism type of the 
P-diagram of the system Ai x Ai. 

7 Basic Characteristic Subgroups 

Let (W, S) be a Coxeter system of finite rank. A basic subgroup (B) of 
(W, S) is said to be reduced if for every set of Coxeter generators 5" of W, 
the base B matches a base B' of (W, S') with 1(5)1 < \(B')\. Note that a 
basic subgroup (B) of (W, S) is unreduced precisely when the base B satisfies 
the conditions of either Theorem 5.2 or 5.4. In particular, every special base 
is unreduced. 
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Let T be a family of finite irreducible Coxeter system isomorphism types. 
Let N(J-) be the normal closure in W of the commutator subgroups of all the 
reduced basic subgroups of (W, S) of isomorphism type contained in T to- 
gether with the commutator subgroups of all the unreduced basic subgroups 
of (W, S) that match a reduced basic subgroup of another system (W, S') of 
isomorphism type contained in T . Let W(T) = W/N^), let r\ : W — * W{F) 
be the quotient homomorphism, and let S^J 7 ) = rj(S). 

Theorem 7.1 The pair (W(J-), S(J-)) is a Coxeter system that can be ob- 
tained from (W, S) be a finite series of elementary edge quotient operations. 
The group N(T) is a characteristic subgroup of W that does not depend on 
the choice of Coxeter generators S . 

Proof: Quotienting out the commutator subgroup of a basic subgroup (B) 
of (W, S) can be realized by reducing all the even labelled edges of the C- 
diagram of {(B), B) to 2 and eliminating all the odd labelled edges of the C- 
diagram. Therefore (W(T), S(T)) is a Coxeter system that can be obtained 
from (W, S) be a finite series of elementary edge quotient operations. 

Let 5" be another set of Coxeter generators of W. By the Basic Matching 
Theorem, iV(jF) is also the normal closure in W of the commutator subgroups 
of all the reduced basic subgroups of (W, S') of isomorphism type contained 
in T together with the commutator subgroups of all the unreduced basic 
subgroups of (W, S) that match a reduced basic subgroup of isomorphism 
type contained in T . Thus N(J-) is a characteristic subgroup of W that does 
not depend on the choice of Coxeter generators S. □ 

We call a subgroup of W of the form A^(jF) a basic characteristic subgroup. 

Corollary 7.2 If W is a finitely generated Coxeter group and N(T) is a 
basic characteristic subgroup of W , then the isomorphism type of W{!F) = 
W/N(F) is an isomorphism invariant ofW. 

8 The Spherical Rank Two Invariant 

B. Miihlherr [7] has announced a solution of the isomorphism problem for 
finitely generated Coxeter groups W such that W has no basic subgroups 
of rank greater than 2 with respect to some set of Coxeter generators. By 
the Basic Matching Theorem, if W has no basic subgroups of rank greater 
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than 2 with respect to some set of Coxeter generators, then W has no basic 
subgroups of rank greater than 2 with respect to every set of Coxeter gen- 
erators. Therefore it makes sense to say that W has no basic subgroups of 
rank greater than 2 without regard to a set of Coxeter generators. 

In this section we describe a characteristic subgroup N 2 of a finitely gen- 
erated Coxeter group W such that W 2 = W/N 2 is a Coxeter group with no 
basic subgroups of rank greater than 2 and such that the isomorphism type 
of W 2 is an isomorphism invariant of W. 

Let X n be one of the finite irreducible Coxeter systems A n , B n , C n , E 6 , 
E7, E 8 , F 4 , G3, G4 of rank n > 3. We now define a characteristic subgroup 
iV(X n ) of X n for each X n . Let iV(X n ) be the commutator subgroup of the 
Coxeter group X n if n > 5 or if X„ = A 4 , G 3 , or G 4 . 

Let ai, a 2 , a 3 be the Coxeter generators of A 3 indexed so that m(ai, a 2 ) = 
nT'(a2,a 3 ) = 3. Let iV(A 3 ) be the normal closure in the group A 3 of the 
element a\a 3 . Then iV(A 3 ) is a characteristic subgroup of A 3 characterized 
by the property that iV(A 3 ) is the unique normal subgroup of A 3 such that 
A 3 /iV(A 3 ) is isomorphic to A 2 according to Table 3 of Maxwell [U]. 

Let bi,b 2 ,b 3 , 64 be the Coxeter generators of B 4 indexed so that 

m(foi, 64) = m(b 2 , 64) = m(6 3 , 64) = 3. 

Let iV(B 4 ) be the normal closure in the group B 4 of the elements bib 2 
and b 2 b 3 . Then iV(B4) is a characteristic subgroup of B4 characterized by 
the property that iV(B 4 ) is the unique normal subgroup of B 4 such that 
B 4 /A^(B 4 ) is isomorphic to A 2 according to Table 3 of Maxwell jSj. 

Let Ci,c 2 ,c 3 be the Coxeter generators of C 3 such that m(ci,c 2 ) = 3 
and m(c 2 , c 3 ) = 4. Let iV(C 3 ) be the normal closure in the group C 3 of the 
element (c 2 c 3 ) 2 . Then iV(C 3 ) is a characteristic subgroup of C 3 characterized 
by the property that iV(C 3 ) is the unique normal subgroup of C 3 such that 
C 3 /iV(C 3 ) is isomorphic to A 2 x A 4 according to Table 3 of Maxwell jHj. 

Let ci, c 2 , c 3 , c 4 be the Coxeter generators of C 4 indexed so that 

m (ci, c 2 ) = m(c 2 , c 3 ) = 3 and m(c 3 , c 4 ) = 4. 

Let iV(C4) be the normal closure in the group C4 of the element Cic 3 . Then 
A^(C 4 ) is a characteristic subgroup of C 4 characterized by the property that 
iV(C4) is the unique normal subgroup of C4 such that C4/iV(C4) is isomor- 
phic to A 2 x Ai according to Table 3 of Maxwell P • 
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Let /i, f 2 , fs, fi be the Coxeter generators of F 4 indexed so that 

m(f h f 2 ) = m(f 3 , f 4 ) = 3 and m(f 2 , / 3 ) = 4. 

Let iV(F 4 ) be the normal closure in the group F 4 of the element ^fs) 2 - 
Then iV(F 4 ) is a characteristic subgroup of F 4 characterized by the property 
that iV(F 4 ) is the unique normal subgroup of F 4 such that F 4 /iV(F 4 ) is 
isomorphic to A2 x A2 according to Table 3 of Maxwell jH] . 

Let (W, S) be a Coxeter system of finite rank. Let N(W) be the normal 
closure in W of the subgroups N((B)) defined above for every base B of 
(W, S) of rank greater than 2. Let = W/N(W). Let rj : W -> be 
the quotient homomorphism, and let = r](S). 

Theorem 8.1 The pair {W^ 2 \ S^) is a Coxeter system that can be obtained 
from (W, S) be a finite series of elementary edge quotient operations. The 
group N(W) is a characteristic subgroup of W that does not depend on the 
choice of Coxeter generators S . 

Proof: Quotienting out the group N((B)) for each base B of (W, S) of 
rank greater than 2 can be realized by elementary edge quotient operations. 
Therefore (W^ 2 \ S^) is a Coxeter system that can be obtained from (W, S) 
be a finite series of elementary edge quotient operations. 

Let 5" be another set of Coxeter generators of W. By the Basic Matching 
Theorem and the characteristic properties of the groups iV(X n ), the group 
N(W) defined in terms of the generators S is the same as the group N(W) 
defined in terms of the generators S' . Thus N(J-) is a characteristic subgroup 
of W that does not depend on the choice of generators S. □ 

Corollary 8.2 IfW is a finitely generated Coxeter group, then the isomor- 
phism type of W {2) = W/N(W) is an isomorphism invariant ofW. 

It may happen that (W^, S^) has a base of rank greater than 2. To 
get a quotient system with no bases of rank greater than 2, we may have 
to quotient out N(W^), and then perhaps repeat the above quotienting 
process several times. This leads to a finite nested sequence 

{1} = N m (W) C N {2 \W) c • ■ ■ C N^(W) 
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of characteristic subgroups of W such that if W^> = W/N^{W) and if 
r\i :W — > is the quotient homomorphism, then 

N^ i+1 \\V) = v r 1 (N{W {l) )) 

for each % = 1, ...,£ — 1, and has no basic subgroups of rank greater 
than 2, and £ is as small as possible. We have that W^ l+r) = (WW)( 2 ) for 
each i = 1, ...,£— 1. Therefore the isomorphism type of for each 
i = 1, . . . ,£ is an isomorphism invariant of W. It follows from the Basic 
Matching Theorem that I does not depend on a choice of Coxeter generators 
for W, and so I is an isomorphism invariant of W. We call I the spherical 
rank 2 class of W. We have £ > 1 with £ = 1 if and only if V4 7 has no 
basic subgroups of rank greater than 2. Figure 2 shows the P-diagrams of a 
sequence . . . , with £ = 4 for the Coxeter group 14 7 = W^. 

Define N 2 = N^(W). Then N 2 is a characteristic subgroup of W such 
that 1^2 = W/N 2 has no basic subgroups of rank greater than 2. The iso- 
morphism type of W 2 is an isomorphism invariant of W which we call the 
spherical rank 2 isomorphism invariant of W . 

Let 7] : W — > W 2 be the quotient homomorphism, and let S 2 = rj(S). 
Then (W 2 , S 2 ) is a Coxeter system that can be obtained from (W : S) by a 
finite series of elementary edge quotient operations. 
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9 Conclusion 



Let (W, S) be a Coxeter system of finite rank. In this paper, we have de- 
scribed three characteristic subgroups N b , N e , N 2 of W each leading to a 
quotient isomorphism invariant of W. It is interesting to note that 

N 2 C N e C N b , 

and so the quotient isomorphism invariants corresponding to N e , N 2 are 
progressively stronger. The algorithm for finding a P-diagram for the system 
(Wb, Sb) starting from a P-diagram of (W, S) is computational fast. 

The algorithm for finding a P-diagram for the system (W e , S e ) is slower 
since it has to determine the bases of (W : S) of type D 2 (4g + 2) that satisfy 
the conditions of Theorem 5.4; but, this algorithm is only slightly slower since 
the conditions of Theorem 5.4 are easy to check. The algorithm for finding a 
P-diagram for the system (W e , S e ) would most likely be incorporated into an 
efficient computer program that determines if two finite rank Coxeter systems 
have isomorphic groups, since the even isomorphism invariant would usually 
determine that two random finite rank Coxeter systems have nonisomorphic 
groups. 

The algorithm for finding a P-diagram for the system (W 2 , S 2 ) is the 
slowest, but it is not much slower, since it only has to find a subdiagram of 
the P-diagram of (W : S) of type A 3 , C 3 or G 3 before it performs an edge 
quotient operation on an edge of the subdiagram. If the subdiagram is of 
type A 3 or G 3 , then the edge with label 2 is eliminated. If the subdiagram 
is of type C 3 , then the 4 edge label is reduced to 2. The algorithm then 
repeats the routine of searching for a subdiagram of type A 3 , C 3 or G 3 and 
performing the corresponding edge quotient operation. 

The algorithm for finding a P-diagram for the system (W 2 , S 2 ) would most 
likely be useful in an efficient program that determines if two finite rank Cox- 
eter systems have isomorphic groups, since the solution of the isomorphism 
problem for finite rank Coxeter systems that have no bases of rank greater 
than 2 is considerably simpler than any general solution of the isomorphism 
problem. 
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